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Overview 


The Communities In Schools (CIS) Model of Integrated Student Supports aims to reduce dropout 
rates by providing students with integrated and tiered support services based on their levels of need. 
The model includes preventive services that are available to all students (Level 1 services) as well as 
intensive, targeted, and sustained services provided through case management (Level 2 services) for 
the 5 percent to 10 percent of students who display significant risk factors for dropping out, such as 
poor academic performance, high absentee rates, or behavioral problems. The CIS model posits that 
these tiered, integrated services will give students the skills and resources they need to succeed, 
which will lead to improvements in their outcomes. 

In elementary schools, the CIS model focuses on improving attendance rates by engaging parents. In 
middle schools, the model begins to emphasize helping students improve their behavior. In high 
schools, the model focuses on services specifically intended to prevent students from dropping out, 
to help them progress through school, and to make sure they graduate. 

This study, which is based on a quasi-experimental research design, examines the CIS model’s 
effect on students’ outcomes in elementary schools, middle schools, and high schools. The sample 
for this study includes 53 CIS schools in Texas and North Carolina (14 high schools, 15 middle 
schools, and 24 elementary schools) that started implementing the CIS model between 2005 and 
2008. The study compares these CIS schools with 78 matched comparison schools (18 high schools, 
24 middle schools, and 36 elementary schools). It is funded by the Edna McConnell Clark Founda- 
tion’s Social Innovation Fund. 

For the high schools, the main finding is that on-time graduation rates increased — and dropout rates 
decreased — in the study schools after the CIS model was launched. Graduation and dropout rates 
also improved in the comparison schools, so it is unclear whether the CIS model was more effective 
than the strategies used by the comparison schools. The findings do suggest that the CIS model may 
be at least as effective as these other approaches. In elementary schools, attendance rates (a central 
outcome the CIS model aims to effect in the elementary grades) improved in schools implementing 
the CIS model more than they did in a group of similar, comparison schools. There was no effect on 
attendance in middle and high schools. In middle schools, English/language arts test scores did not 
improve in schools implementing the CIS model, whereas they did improve in a group of similar, 
comparison middle schools. There was no effect on test scores in elementary and high schools. (It is 
important to note, however, that the CIS model does not attempt to improve state test scores or the 
quality of instruction provided during regular school hours.) Unfortunately, it was not possible to 
evaluate whether the CIS model improved middle school students’ behavioral outcomes, which is 
the model’s primary goal in those grades. 




Preface 


Although many services and other fonns of support are available to students at risk of dropping 
out, far too many students still fail to complete high school. The problem may lie in the fact that 
services for at-risk students are spread across many different government agencies and nonprofit 
organizations, which makes it more challenging for schools to identify the services available to 
students and for students themselves to use them. 

This report examines an integrated model of student support created by Communities In 
Schools (CIS), which is now working in about 2,400 schools and 360 school districts. The CIS 
model provides comprehensive and integrated services to students in different areas (academics, 
behavior, social skills and life skills, family outreach, health and wellness, etc.), delivered with 
varying intensity and duration based on students’ level of need. Level 1 short-term, preventive 
services are broadly available to all students at a school, whereas Level 2 intensive, longer-term, 
targeted services are for students at higher risk of dropping out. 

The schools in this study, located in Texas and North Carolina, started implementing 
the CIS model between 2005 and 2008. Using a quasi-experimental design, the study found 
mixed but promising results. In elementary schools, it appears that the CIS model may have 
improved attendance rates, which is consistent with the findings of other studies of CIS (includ- 
ing a randomized experiment in K-8 schools in Chicago). In middle schools, the CIS model 
does not appear to have improved any of the outcomes that could be measured in this study 
(attendance rates and test scores). In high schools, graduation rates improved after the CIS 
model was implemented, but it is not clear whether CIS caused these improvements. 

Thus far, two quasi-experimental studies — the present one by MDRC and an earlier 
one by ICF International — have found that high schools implementing the CIS model have 
increased their graduation rates. In both studies, however, these findings have been inconclusive 
because of limitations related to finding a group of credible comparison schools. Therefore, a 
useful next step for CIS would be an evaluation based on a school-level randomized experi- 
ment. Such an experiment would provide the most rigorous evidence of the model’s effects, and 
ultimately the most useful information for policymakers, districts, and schools. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Every day more than 7,000 students drop out of school. 1 One- fifth of students who enter high 
school do not graduate within four years, 2 and more than two-fifths of Latino and African- 
American boys drop out. 3 Many students at risk of dropping out need academic and social 
services and other fomis of support to make it through high school. However, these services are 
scattered across numerous government agencies and nonprofit organizations, which limits their 
potential to change the path of an at-risk student. Integrating student support services and 
connecting them with schools is viewed as a promising approach to assist school staff members 
and help students stay on track to graduate. 4 

This report presents the findings from a quasi-experimental study of the Communities 
In Schools (CIS) Model of Integrated Student Supports (referred to in this report as the “CIS 
model”), a promising whole-school approach that aims to reduce dropout rates by providing 
students with integrated and tiered support services based on their needs. Because some of the 
factors that put students on the path toward dropping out of high school are established well 
before ninth grade, CIS works with elementary schools, middle schools, and high schools. This 
study of the CIS model, which is funded by the Edna McConnell Clark Foundation’s Social 
Innovation Fund, provides a unique opportunity to gain a better understanding of the potential 
effects of a tiered and integrated approach to dropout prevention that serves students of all 
ages. 5 


In high schools, the main finding from this study is that on-time graduation rates in- 
creased — and dropout rates decreased — in study schools after the CIS model was launched. 
However, it is not clear whether these improvements were caused by the CIS model. On the one 


'Christopher B. Swanson, “Progress Postponed,” Education Week 29, 34 (2010): 22-23. 

2 Richard J. Mumane, “U.S. High School Graduation Rates: Patterns and Explanations,” Journal of 
Economic Literature 51, 2 (2013): 370-422; Marie C. Stetser and Robert Stillwell, Public High School Four- 
Year On-Time Graduation Rates and Event Dropout Rates: School Years 2010-11 and 2011-12 (Washington, 
DC: National Center for Education Statistics, U.S. Department of Education, 2014). 

3 Schott Foundation for Public Education, The Urgency of Now: The Schott 50 State Report on Public 
Education and Black Mcdes (Cambridge, MA: Schott Foundation for Public Education, 2012). 

4 Kristin A. Moore, Selma Caal, Rachel Carney, Laura Lippman, Weilin Li, Katherine Muenks, David 
Murphey, Dan Princiotta, Alysha Ramirez, Angela Rojas, Renee Ryberg, Hannah Schmitz, Brandon Stratford, 
and Mary Terzian, Making the Grade: Assessing Evidence for Integrated Student Supports (Bethesda, MD: 
Child Trends, 2014). 

5 The Social Innovation Fund is a program of the Corporation for National and Community Service. The 
Social Innovation Fund combines public and private resources to increase the impact of innovative, community- 
based solutions that have compelling evidence of improving the lives of people in low-income communities 
throughout the United States. 
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hand, graduation and dropout rates improved by greater amounts in the CIS high schools than in 
a group of comparison high schools, which suggests that the CIS model may have improved 
these rates more than they would have improved otherwise. On the other hand, the comparison 
schools and CIS schools had different graduation and dropout rates before the CIS model was 
implemented, so the comparison schools may not provide a credible reference point. For this 
reason, it is not possible to determine whether the CIS model was more effective than the 
strategies used by the comparison schools, although the study’s findings suggest that the CIS 
model may be at least as effective as these other approaches. 

In elementary schools, attendance rates (a central outcome measure CIS aims to affect 
in the elementary grades) improved in schools implementing the CIS model more than they did 
in a group of similar comparison schools. In middle schools, English/language arts (ELA) test 
scores did not improve in schools implementing the CIS model, whereas they did improve in a 
group of similar, comparison middle schools. Unfortunately, it was not possible to evaluate 
whether the CIS model improved middle school students’ behavioral outcomes, which is the 
model’s primary goal in those grades. 

What Is the CIS Model? 

Communities In Schools, which was founded in 1977 by children’s advocate Bill Milliken, 
works with low-income K-12 students who are at risk of failing or dropping out of the nation’s 
poorest-perfonning schools. CIS seeks to reduce dropout rates by integrating preventive 
services available to the entire school with intensive, targeted, and sustained services for the 5 
percent to 10 percent of students who display significant risk factors for dropping out, such as 
poor academic performance, high absentee rates, or behavioral problems. CIS now serves 1.5 
million students and their families in 25 states and the District of Columbia. It is active in 
approximately 2,300 schools and 360 school districts. 6 

CIS provides services and support to students in 10 different areas: academics, behav- 
ior, social skills and life skills, basic needs and resources, college and career preparation, 
enrichment and motivation, family outreach and engagement, health and physical wellness, 
community service, and mental health. These 10 categories of services are provided at two 
levels of intensity and duration, depending on students’ needs: 

• Level 1 school-wide and preventive services: Level 1 services are broadly 
available to all students at the school and are usually short-term, low- 
intensity activities or services (for example, making clothing or school sup- 


6 Communities In Schools, 2015 Annual Report (Arlington, VA: Communities In Schools, 2015). 
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plies available to students, organizing a school-wide career fair, or hosting a 
financial aid workshop for twelfth-graders). 

• Level 2 targeted services: In contrast, Level 2 services are intensive, often 
long-term, and targeted forms of support that are delivered to students who 
are displaying one or more significant risk factors for dropping out, such as 
poor academic performance, a high absentee rate, or behavioral problems. 

Level 2 services include forms of support such as individual and group coun- 
seling, tutoring, and after-school programs. 

The CIS model’s goals are different in high schools, middle schools, and elementary 
schools. In all three, the CIS model provides support in each of the 10 service categories, but the 
emphasis in each case is calibrated to the model’s core goals for that type of school. In elemen- 
tary schools, the model focuses on improving attendance rates by reaching out to and engaging 
parents. In middle schools, the model begins to emphasize helping students improve their 
behavior. In high schools, the model focuses on services specifically intended to prevent 
students from dropping out, help them progress through school, and make sure they graduate. 

How Was the CIS Model Evaluated? 

This study conducted by MDRC examines whether introducing the CIS model with all of its 
components improves schools’ graduation rates, dropout rates, attendance rates, and state test 
scores. 


The effect of the CIS model is evaluated using a comparative interrupted time series 
(CITS) design. 7 Two groups of schools are studied: one group that implemented the CIS model 
(CIS schools) and another group that did not implement the CIS model but was free to adopt 
some other refonn or initiative (comparison schools). The first step in a CITS design is to 
determine the trends in school outcomes for the CIS schools and the comparison schools during 
the years before the intervention was launched. These are called the “baseline trends.” The 
second step is to gauge how much the CIS and comparison schools “deviated” from their 
baseline trends after the intervention was launched. The outcomes of the two groups of schools 
are not compared directly; instead, the analysis compares the amount by which the two groups 
deviated from their separate baseline trends. If the CIS model is more effective than other 
programs or reforms available to schools (those used by the comparison schools), then the CIS 


7 For a discussion and history of CITS designs, see William R. Shadish, Thomas D. Cook, and Donald T. 
Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Boston: 
Houghton Mifflin, 2002). For a discussion of these designs in the context of education research, see Howard S. 
Bloom, “Using ‘Short’ Interrupted Time-Series Analysis to Measure the Impacts of Whole-School Reforms, 
with Applications to a Study of Accelerated Schools,” Evaluation Review 27, 3 (2003): 3-49. 
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schools should experience improvements relative to their baseline trend that exceed the im- 
provements found in comparison schools. 

The comparison schools play an important role in this study design. Their trends over 
time reveal what happened to similar schools that did not implement the CIS model and thus are 
intended to show how much CIS schools’ outcomes would have improved had they not imple- 
mented it. For example, had they not implemented the CIS model, CIS schools’ outcomes may 
have improved because they chose to implement some other school reform model (instead of 
CIS), or they may have improved due to a district-wide or statewide policy change. The CITS 
design makes it possible to identify the CIS model’s effect over and above the effect of these 
alternative reforms and system-wide policy changes (that is, the model’s net effect). 

This study examined 53 schools in Texas and North Carolina (14 high schools, 15 mid- 
dle schools, and 24 elementary schools) that started implementing the CIS model from 2005 to 
2008. The study sample also includes 78 comparison schools (18 high schools, 24 middle 
schools, and 36 elementary schools). The comparison schools were chosen from non-CIS 
schools in counties in Texas and North Carolina where there is a CIS presence, ensuring that 
they are located in the kinds of districts where CIS typically operates. Matching methods were 
used to select comparison schools whose baseline characteristics and trends in school outcome 
measures were as similar as possible to those of the CIS schools. 

Overall, the CIS schools and the comparison schools have similar baseline characteris- 
tics and outcome measure values, with one exception: The comparison high schools had higher 
graduation rates than the CIS schools (by about 9 percentage points) and lower dropout rates 
(by about 1 .5 percentage points). These differences are small enough to meet commonly used 
criteria for baseline equivalence, such as the one used by the What Works Clearinghouse. 8 
Substantively, however, these differences are large enough that the CIS schools may have had 
more incentive to turn themselves around. For example, had they not implemented the CIS 
model, the CIS schools may have chosen to implement different — but still intensive — whole- 
school interventions to improve their low graduation rates. In contrast, the comparison schools 
may have felt less need or pressure than the CIS schools to initiate a turnaround, and according- 
ly, they may have adopted less intensive strategies to improve their graduation rates. In that 
case, the comparison schools’ deviations from trend would not provide the right information 
about what would have happened to the CIS schools had they not implemented the model. Any 
improvements in the comparison schools’ graduation rates would understate the amount CIS 


8 The What Works Clearinghouse criterion is that differences in baseline characteristics should not exceed 
0.25 standard deviations. See What Works Clearinghouse, What Works Clearinghouse Procedures and 
Standards Handbook, Version 3.0 (Washington, DC: U.S. Department of Education, Institute of Education 
Sciences, 2014). 
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schools’ own rates would have improved without the CIS model; by extension, the findings 
from this study would overstate the true effect of the CIS model. Various sensitivity tests were 
used to explore this possibility as well as other factors that could affect the credibility of the 
comparison schools as a reference point. 

Data for the study were obtained from existing, publicly available school-level histori- 
cal databases. 9 Using these data, this study is able to examine whether the CIS model was 
successful at meeting its main objectives: increasing on-time graduation rates and reducing high 
school dropout rates. The study can also examine whether the CIS model had a positive effect 
on attendance rates (the main outcome of interest in elementary schools) and perfomiance on 
state tests. However, the study is not able to evaluate whether the CIS model improved students’ 
behavioral outcomes, which is one of the model’s main goals in middle schools. Another data- 
related limitation is that infonnation is not available on what kinds of services and support are 
offered to students in the comparison schools, and how these services differ from those included 
in the CIS model. This limitation makes it more challenging to interpret the study’s findings, 
because it is unclear what initiatives and services the CIS model is being compared with. 

In addition to evaluating the effect of CIS’ whole-school model, MDRC is also evaluat- 
ing the effect of one component of the CIS model — Level 2 services — using a student-level 
random assignment research design. This study is being conducted in 28 secondary schools in 
North Carolina and Texas. Eligible at-risk students were randomly assigned to receive Level 2 
case management and services or to continue with business as usual, with access to whatever 
other forms of support were available to them. 

The first report from the random assignment study described the effect of Level 2 ser- 
vices on students’ behavioral and academic outcomes after one year. 10 It found that Level 2 
services had a positive and statistically significant impact on students’ reports of having caring, 
supportive relationships with adults outside of home and school; the quality of their peer 
relationships; and their belief that education has positive value for their lives. But MDRC found 
no evidence that Level 2 services had improved students’ attendance, course performance, or 
behavior. However, it is too early to make any definitive conclusions, because one year of case 
management may not be sufficient to improve the outcomes of these students, all of whom face 
serious academic and personal challenges. Accordingly, the next (and final) report will examine 
the effect of Level 2 case management after two years. 

9 These databases include the Common Core of Data and data sets maintained by the Texas Education 
Agency, the North Carolina Research and Data Center at Duke University, and the North Carolina State 
Department of Education. 

"'William Corrin, Leigh Parise, Oscar Cerna, Zeest Haider, and Marie-Andree Somers, Case Management 
for Students at Risk of Dropping Out: Implementation and Interim Impact Findings from the Communities In 
Schools Evaluation (New York: MDRC, 2015). 
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Did the CIS Model Improve School Outcomes? 

MDRC’s study of the CIS model examines the following two research questions: 

• After launching the CIS model, did the CIS schools have better outcomes 
than predicted by their baseline trend? This question is answered by looking 
at the deviations from baseline trend for the CIS schools in the study. 

• Did the CIS schools’ outcomes improve more than they would have in the 
absence of the CIS model? This question is answered by looking at the dif- 
ference between the deviations from baseline trend for the CIS schools and 
the comparison schools. This difference represents the estimated net effect of 
the CIS model. 

High School Graduation and Dropout Rates 

• After three years of implementing the CIS model, on-time graduation 
rates and dropout rates improved by statistically significant amounts in 
the CIS high schools, relative to what would have been expected given 
their baseline trends. 

• It is not possible to determine the extent to which these improvements 
can be attributed to the CIS model. Graduation and dropout rates im- 
proved more in the CIS schools than in the comparison high schools, but 
this pattern of results depends on the choice of comparison schools. 

Figure ES.l plots the trend in graduation rates for the CIS high schools (solid black 
line) and the comparison schools (solid gray line), during the school years before the CIS model 
was launched, and also shows graduation rates after the model was launched. Before the CIS 
model was implemented, graduation rates had been steadily declining for both groups of 
schools. After the model was launched, graduation rates in the CIS high schools were higher 
than expected relative to these schools’ baseline trend. Graduation rates were also higher than 
predicted in the comparison schools, perhaps because they chose to use a strategy other than the 
CIS model to turn themselves around. However, graduation rates improved more in the CIS 
schools than in the comparison schools, and therefore the estimated net effect of the CIS model 
is positive. 

Table ES.l summarizes the findings. As shown in the first panel, the estimated effect of 
the CIS model on graduation rates is consistently positive during the first three years of 
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Figure ES.l 

Baseline Trends and Deviations from Trend in High School Graduation Rates 



Relative year (CIS introduced at Year 1) 


O Comparison schools' outcomes ^CIS schools' outcomes 

SOURCES: MDRC calculations based on school-level data on student outcomes from the Texas Education Agency. 



Table ES.l 


Estimated Effects on Selected Outcomes 



Deviation from Baseline Trend 




P-Value for 


CIS 

Comparison 

Estimated 

Lower 

Upper 

Estimated 

Outcome and Follow-Up Y ear 

Schools 

Schools 

Effect 

90% Cl 

90% Cl 

Effect 

High schools 

Graduation rate (%) 







Year 1 

2.71 f 

-1.74 

4.44 

-0.48 

9.37 

0.137 

Year 2 

6.19 fff 

0.69 

5.51 

-0.77 

11.78 

0.148 

Year 3 

15.58 fff 

8.08 ff 

7.50 * 

0.27 

14.73 

0.088 

Number of schools 

8 

11 





Dropout rate (%) 







Year 1 

-0.75 

0.30 

-1.06 

-2.40 

0.29 

0.197 

Year 2 

-1.85 ff 

-0.65 

-1.19 

-2.75 

0.36 

0.207 

Year 3 

-3.8 fff 

-2.3 fff 

-1.5 

-3.4 

0.5 

0.211 

Number of schools 

14 

18 





Middle schools 

ELA state test scores (z-scores) a 







Year 1 

0.01 

0.05 f 

-0.05 

-0.11 

0.02 

0.252 

Year 2 

0.02 

0.10 fff 

-0.08 

-0.16 

0.00 

0.110 

Year 3 

0.00 

0.11 fff 

-0.11 * 

-0.20 

-0.01 

0.061 

Number of schools 

8 

15 





Elementary schools 

Attendance rate (%) 







Year 1 

0.30 fff 

0.14 f 

0.16 

-0.07 

0.39 

0.257 

Year 2 

0.44 fff 

0.17 

0.27 

-0.01 

0.54 

0.111 

Year 3 

0.61 fff 

0.20 f 

0.41 ** 

0.10 

0.72 

0.030 

Number of schools 

24 

36 






(continued) 













Table ES.l (continued) 


SOURCES: MDRC calculations based on school-level data on student outcomes from state websites (the Texas 
Education Agency and the North Carolina State Department of Education) and from the North Carolina Education 
Research Data Center at Duke University. 

NOTES: The values in the "CIS Schools" and "Comparison Schools" columns are the estimated deviations from 
baseline trend for each group of schools. The values in the "Estimated Effect" column are the differences between 
CIS schools and comparison schools with respect to their deviations from baseline trend. The values in the "Lower 
90% Cl" and "Upper 90% Cl" columns are the 90 percent confidence intervals for the estimated effects. Rounding 
may cause slight discrepancies in calculating sums and differences. 

A two-tailed test was applied to estimated deviations and estimated differences between CIS schools and 
comparison schools. The statistical significance of estimated deviations is indicated as: f f f = 1 percent; f f = 5 
percent; f = 10 percent. The statistical significance of estimated effects is indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

“State test scores were converted to z-scores based on the estimated student-level mean and standard deviation 
in test scores for the state. 


implementation, and it becomes larger over time, as one would expect. 11 By the third year, the 
estimated effect is 7.5 percentage points, which represents an 11 percent increase in graduation 
rates relative to the last baseline year. An effect of this size means that in the third year of 
implementation, 55 additional students in each study school graduated from high school on 
time. Though the CIS model’s estimated effect on dropout rates is not statistically significant, 
those results follow a similar pattern (as shown in the second panel of Table ES. 1). 

As explained earlier, a central assumption of the CITS design is that the comparison 
schools’ deviations from their baseline trend represents what would have happened to the CIS 
schools in the absence of the intervention. As shown in Figure ES.l and discussed earlier, the 
comparison schools had substantially higher baseline graduation rates (and lower dropout rates) 
than the CIS schools, so they may have had less incentive to turn their graduation rates around. 
This difference places an additional burden on the study to demonstrate that the comparison 
schools can still represent what would have happened to CIS schools in the absence of the CIS 
model. Various sensitivity tests were conducted to examine the credibility of this assumption. 
Some of these sensitivity tests suggest that the comparison schools do provide a good point of 
reference for the CIS schools, but others suggest that they may not, and that the net effects 
shown in Table ES. 1 may therefore be overstated. 

Given these results, it is not possible to determine whether the CIS model improved 
graduation and dropout rates more than they might have improved otherwise, or if it did, by 


"Graduation rates are only available for 8 of the 14 CIS high schools, because data on this outcome are 
only available for high schools in Texas. 
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how much. However, the study does show that the average graduation and dropout rates for the 
CIS high schools improved relative what would have been expected given their prior trends. In 
addition, although it is not possible to determine whether the CIS model was more effective 
than the strategies used by the comparison schools, the study’s findings suggest that the CIS 
model may be at least as effective. 

Elementary School Attendance Rates 

• After three years, the average attendance rate increased by a statistically 
significant amount in the CIS elementary schools, relative to what would 
have been expected given their baseline trend. 

• The improvement in attendance rates for these CIS schools was larger 
than the improvement in comparison schools by a statistically significant 
amount. The CIS model appears to have improved these schools’ at- 
tendance rates more than they would have improved otherwise. 

Attendance is the main student outcome CIS aims to affect in elementary schools. After 
the CIS model was launched, attendance rates in the CIS elementary schools were higher than 
predicted by these schools’ baseline trends by a statistically significant amount (as shown in the 
last panel of Table ES.l). Attendance rates in the comparison schools were also higher than 
predicted, perhaps because these schools were using other strategies to improve their attendance 
rates. However, in all three follow-up years, the CIS elementary schools deviated from their 
baseline trend by a greater amount than in the comparison schools, and that difference becomes 
statistically significant in the third year of implementation. This pattern of results holds across 
all of the sensitivity tests that were conducted to examine the credibility of the comparison 
schools as a reference point. Therefore, the CIS model may have improved attendance rates in 
the study’s elementary schools more than the strategies being used by the comparison schools. 

On its face, the CIS model’s estimated effect on attendance rates may seem small: By 
the third year, the estimated effect is 0.4 percentage points, which represents an extra 0.7 days 
of school. 12 However, it is important to note that attendance rates were already high before the 
CIS model was launched (96.1 percent), so an effect of 0.4 percentage points gets schools 10 
percent closer to perfect attendance. 13 In addition, average daily attendance rates can mask high 
rates of chronic absenteeism: 1 1 percent of elementary school students are chronically absent 


12 This number assumes a school year of 180 days. 

13 The gap between baseline attendance and perfect attendance is 3.9 percentage points (100 - 96.1). An 
effect of 0.4 percentage points is 10 percent of that gap (0.4 3.9). 
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(that is, they are absent 15 or more days during the school year). 14 A small effect on overall 
attendance rates may represent a larger effect on chronic absenteeism. 

Middle and High School Attendance Rates 

In CIS middle schools, attendance rates did not improve by a statistically significant 
amount. In CIS high schools, attendance rates did improve by a statistically significant amount, 
but not by a greater amount than they would have improved otherwise. These results are 
consistent across the sensitivity tests that were conducted to examine the credibility of the 
comparison schools. 

Middle School State Test Scores 

• After three years of implementation, state test scores did not improve in 
the CIS middle schools, relative to what would have been expected given 
their baseline trends. 

• During the same period, state test scores did improve by a statistically 
significant amount in the comparison schools. CIS middle schools’ test 
scores appear to be lower than they would have been otherwise. 

After the CIS model was launched, state test scores in the CIS middle schools were not 
better than predicted by their baseline trends. In contrast, the comparison schools performed 
better on state tests than predicted, perhaps because they implemented other reforms or initia- 
tives to improve their students’ performance. As a result, the CIS model’s estimated effect on 
middle school students’ ELA state test scores is consistently negative and it becomes more 
negative over time (as shown in the third panel of table ES. 1). 15 By the third year, the estimated 
effect size is -0.11 standard deviations, which translates into about 14 weeks of learning. 16 
These results hold across all of the sensitivity analyses that were conducted to examine the 
credibility of the comparison schools as a reference point. This result suggests that the CIS 
model may have been less successful at improving state test scores than the strategies used by 


14 Office of Civil Rights, 2013-2014 Civil Rights Data Collection, a First Look: Key Data Highlights on 
Equity and Opportunity Gaps in our Nation’s Public Schools (Washington, DC: U.S. Department of 
Education, Office of Civil Rights, 2016). 

15 The number of CIS and comparison schools for this analysis is less than the total number of middle 
schools in the study, because time-series data on state test scores are not available for all schools. 

16 The conversion from effect size to weeks of learning is based on data in Carolyn J. Hill, Howard S. 
Bloom, Alison Rebeck Black, and Mark W. Lipsey, “Empirical Benchmarks for Interpreting Effect Sizes in 
Research,’’ Child Development Perspectives 2, 3 (2008):172-177. Middle school students make gains of about 
0.008 standard deviations per week in reading. Therefore, an effect size of -0.11 standard deviations is 
equivalent to 14 weeks of learning (0.11 -=- 0.008). 
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the comparison schools. It is important to note, however, that the CIS model does not attempt to 
improve state test scores or the quality of instruction provided during regular school hours. 

Elementary School and High School State Test Scores 

In CIS elementary schools, ELA state test scores improved by a statistically significant 
amount, but not by more than they would have improved otherwise. These results are consistent 
across the sensitivity tests that were conducted to examine the credibility of the comparison 
schools. In CIS high schools, ELA state test scores also improved by a statistically significant 
amount. However, it is not possible to determine the extent to which these improvements can be 
attributed to the CIS model due to the limitations in identifying a group of credible comparison 
schools described above. 

Discussion 

Because this study is based on a small number of purposefully selected schools that started 
implementing the CIS model a decade ago, its findings may not represent the effect of the CIS 
model nationally as it exists today. Therefore, the results from this study should be considered 
alongside the results of other evaluations of the CIS model, two of which are worth noting. The 
first is a national quasi-experimental study of the CIS model conducted by ICE International. 17 
The second study is an (as yet) unpublished school-level random assignment evaluation of the 
CIS model in Chicago K-8 schools. 18 

All three studies of the CIS model conducted thus far have found positive effects on at- 
tendance rates for younger students. With respect to state test scores, the effect of the CIS model 
appears to depend on the local context: This study finds negative effects in middle schools, 
whereas the Chicago study finds positive effects. This difference suggests that in some settings, 
the CIS model can have positive effects on students’ test scores and be more effective than the 
other strategies available. 

The findings in high schools are promising but more difficult to interpret. The ICE 
study finds an improvement of 1.7 percentage points in ninth-grade students’ probability of 
graduating from high school, but this effect is not statistically significant. In the present study, 
graduation rates improved for the CIS schools after they launched the model, but it is unclear to 


17 1CF International, Communities In Schools National Evaluation Volume 1: School-Level Report. Results 
from the Quasi-Experimental Study, Natural Variation Study, and Typology Study (Fairfax, VA: ICF 
International, 2008). 

ls David N. Figlio, “Experimental Evidence of the Effects of the Communities In Schools of Chicago 
Partnership Program on Student Achievement,” Northwestern University Working Paper (Evanston, IL: 
Northwestern University, 2015). 
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what extent graduation rates would have improved had these schools not implemented the CIS 
model. 


Therefore, it would be a useful next step for CIS to undertake a rigorous and large-scale 
study of its model in high schools in particular, based on a randomized experiment and supple- 
mented by a cost study. A school-level random assigmnent research design would provide the 
best evidence of the CIS model’s effect on student outcomes relative to that of other programs 
and strategies. By collecting cost data on the implementation of the CIS model as well as cost 
data on the strategies and interventions used by the control schools, one could also determine 
the CIS model’s relative cost-effectiveness, which would ultimately provide the most useful 
information for school districts. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 
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